Last year, we featured a lengthy interview with tech journalist/science fiction author Cory Doctorow about his book, Enshittification: Why Everything Suddenly Got Worse and What To Do About It. The prolific Doctorow is back with a provocative new book that serves as a follow-up of sorts, focusing on AI and related issues: The Reverse Centaur's Guide to Life After AI.

Doctorow doesn't actually enjoy talking about AI, but he's constantly being asked to comment on it. "I made the tactical error of being sick of talking about AI," Doctorow told Ars. "So I wrote a book about why I think it's a dumb thing to keep asking people to talk about, and now I have to talk about it." Reverse Centaur is Doctorow's attempt to "sort out the bullshit from the material reality."

In automation theory, per Doctorow, a "centaur" describes a human augmented with a technology, like machine learning, or even just driving a car or using autocomplete. A reverse centaur "is a machine head on a human body, a person who is serving as a squishy meat appendage for an uncaring machine," Doctorow said in a speech last December. He gave the example of an Amazon delivery driver, surrounded by AI cameras monitoring their driving, who essentially serves as a peripheral to the delivery van.

Being a centaur is generally viewed as a positive thing; few people relish being a reverse centaur. And yet the AI industry seems intent on using those tools to create more reverse centaurs. It's one thing to incorporate AI tools into the medical field to help radiologists process X-ray images and spot potential tumors they might otherwise miss. It's quite another to fire nine out of 10 radiologists and let AI make the diagnoses, with the remaining radiologist solely responsible for checking the AI's work—and, ultimately, taking the blame for any errors.

Doctorow is not virulently anti-AI; he uses AI tools regularly and sees potential in many of those tools as useful plugins or cool new apps. But he is nonetheless alarmed at all the hype surrounding AI, the enormous capital expenditures, the unrealistic expectations and self-serving messaging, and the potentially catastrophic economic consequences when the AI bubble inevitably pops.

"The bubble doesn't want cheap useful things," Doctorow said. "It wants expensive 'disruptive' things: big foundational models that lose billions of dollars every year. When the AI investment mania halts, most of the models are going to disappear, because it just won't be economical to keep the data centers running. The collapse of the AI bubble is going to be ugly. Seven AI companies currently account for more than a third of the stock market, and they endlessly pass around the same $100 billion IOU. AI is the asbestos in the walls of our technological society, stuffed with wild abandon by a finance sector and tech monopolists run amok. We will be excavating it for a generation or more."

Naturally, Doctorow has some ideas about how to push back against the prevailing narrative of AI's inevitability. Ars caught up with him to learn more.

Ars Technica: We touched briefly on AI last year when we chatted about your prior book. Reverse Centaur seems like a natural outgrowth of that.

Cory Doctorow: Enshittification is primarily a thesis about how firms in the absence of constraint get tilted to the bad, but it's also a thesis about how the constraint of competition, when it falls away, produces all kinds of perverse outcomes. One of those perverse outcomes is that firms that have saturated their markets can no longer grow, and they have to find other markets. There's a ticking bomb when you saturate your market because it's only a matter of time until investors start to worry that you're not a growth stock, you're a mature stock. Mature stocks trade at a small fraction of the multiple that growth stocks do.

There's an enormous amount of liquidity in growth stocks, which means that you can use growth stocks to grow. You can buy other companies with shares, and shares are an endogenous substance that you make on the premises by typing zeros into a spreadsheet. Firms with growth stocks can grow by typing zeros, whereas firms that are mature, they have to use money if they want to grow, and you're not allowed to make money on the premises. If you do, the Treasury Department shows up and takes you away in handcuffs. So you can see why firms would be very anxious to maintain the perception that they have room for growth even after they have 90 percent market shares.

That's why those firms started promoting stories about how they were going to conquer imaginary markets. Imaginary markets have no agreed-upon valuation because you just made them up. Unless you can turn an imaginary market into a real market pretty quickly, you need to come up with another imaginary market and announce that this is the new imaginary market you're going to conquer. It's easier than you'd think because the capital markets have the object permanence of a toddler, and they would lose a game of peekaboo if they were drafted to play in the league. So you can say, "Oh, actually, it's not metaverse. It's crypto. It's not crypto. It's Web3. It's not Web3. It's something else." And the markets will forgive you, provided you do it quickly enough.

But something different happened with AI. It is much, much bigger in terms of capitalization than anything we've ever seen—not just bigger than other tech bubbles, bigger than other bubbles. When I wrote the book, capital expenditure (CapEx) globally was $700 billion, now it's $1.4 trillion. Meta wasted $60 billion on the metaverse. They spent $150 billion in the last three years on AI, and they say they're going to spend another $150 billion this year.

So this is a much bigger bet, and it raises the question: If the material basis for this is creating a narrative so that you can continue to grow by dint of having a highly liquid growth stock, what's the ideological basis? Why are people willing to make such a bigger bet? Some of it is that there's more "there" there with AI. It's real computer science. It was remarkable 10 years ago, when a couple of computer scientists and their grad students took some existing techniques, applied them in a new way, and got a very surprising result that turned out to not only produce dividends the first time around, but to have somewhat linear returns on investment, which is not usually the case.

There was a lot of low-hanging fruit in AI, although it's tapering off now because, as they say in finance, anything that can't go on forever has to stop. So we're losing the end of that growth period in terms of returns to scale.

Ars Technica: Why do you think AI is so appealing to political and business leaders in particular? 

Cory Doctorow: It's not just that it makes for a good demo. AI really appeals to a fantasy that I think all of us have to some extent but that powerful people really have, of a world without people in it—because hell really is other people. You can't get stuff done without other people helping you. You can't have romance without a romantic partner. You can't have social media without people to socialize with. You can't play a board game, or do a startup, or build a bridge, or build a house, or do politics without other people. And other people stubbornly refuse to organize everything they do to make you happy.

Particularly if you're rich and powerful, it's very galling. So AI is very attractive. One of the reasons DOGE fired so many government workers was because it played into the fantasy that you can have a government without government employees. In the corporate sphere, it's the fantasy of a business without workers, because every corporate leader is haunted by the secret fear that if they don't show up for work, everything goes on just fine. But if the workers don't show up, everything shuts down. Maybe they're not really driving the car, maybe they're strapped in the backseat with a toy steering wheel.

If that's the case, AI will let them wire the toy steering wheel directly into the drivetrain. So you can have an amazing idea as a corporate visionary, and you don't have to have any ego-shattering confrontations with people who know how to do things, who tell you you're actually an idiot. You just type some stuff to the chatbot, and it shits out your product. If you combine those two things—the material necessity to have a growth narrative and the ideological attractiveness of a world without people—you get $1.4 trillion in CapEx for a sector that is turning over $50 billion a year and has to replace all of its assets every 24 to 30 months.

Ars Technica: You raised an interesting point recently on your blog: Workers actually wanted earlier technological breakthroughs and often had to fight to get them into the workplace. With AI, people are more likely to feel that the technology is being shoved down our throats; some workers are even required to use it.

Cory Doctorow: I think that's entirely right. One of the things that I've been attending to a lot lately is the difference between the bubbles that we had before and the bubble that we're having now. People will say, "Oh, Amazon wasn't profitable, and it became profitable. And the web wasn't profitable, and it became profitable. The web was a bubble." Of course the web was a bubble. You don't get pets.com and all those Super Bowl ads without a bubble. But it is a very obvious error of logic to say, "Once, there was a thing that lost money and then it made money, therefore, if you are losing money, someday you'll make money."

The thing that made the web profitable was not that it was unprofitable; it was things like good unit economics, where every time someone started using the web, the web got less unprofitable. Every time a web user used the web again, the total profits generated went up. Every generation of web technology made the web more profitable. That's the opposite of AI. Every AI customer loses money for the company, every use of AI by that customer loses money for the company, and every generation of AI loses more money than the last one. AI is the money-losingest thing our species has ever done. We have never lost as much money as we've lost on AI.

Another giant material difference is the social reception. If you look back to the business press of the aughts and the late '90s, it's full of hand-wringing editorials about how bosses will cope with workers who are smuggling in the web. You look at those same press outlets today, and it's full of people saying, "What are we going to do about the fact that no one in the workplace wants to use AI?"—along with ads for firms that will spy on your workers for you so that you can punish the workers who refuse to use AI.

Ars Technica: AI nonetheless does have thoughtful, sensible defenders.

Cory Doctorow: One of the paradoxes that I try to explore in this book is the workers who are not fools, who are historic good, reliable narrators of their own experience, and who tell you that AI is making their lives better. The foundational idea of science fiction is that what the gadget does is less important than who it does it for and who it does it to. I call those people centaurs. They are workers who are assisted by technology and who decide how that technology is going to assist them. Whereas the workers who hate it are workers who are being asked to produce more with AI at the expense of quality, at a higher speed, at the expense of their own wellbeing, and who understand that they're being recruited to be what Dan Davies calls accountability sinks—to take the blame when the AI screws up their job.

Once you put it that way, it's very easy to see why some workers would say, "Oh yeah, I found a thing that AI is good for and I use it, and that's fine. I'm even excited about it." And why other workers would be like, "This is making me miserable." It's the difference between the words on the Greek temple, "Know thyself," and your boss shining 16 cameras in your face and going, "I know you better than you do. And by the way, I think you could work an extra hour a day without breaking a sweat."

Ars Technica: You make a point of emphasizing that you are not fundamentally anti-AI, despite sharply criticizing the industry. 

Cory Doctorow: I have many comrades who describe themselves as anti-AI, and I've had some very spirited, productive, but heated debates with those people because I don't think AI is exceptional. That means that I don't think it's exceptionally evil. The argument that it's the fruit of the poisonous tree, that it was made by bad people in bad ways, so you shouldn't use it—I think it's very foolish. That is not the merit on which we judge technology.

You can talk about whether giving money to these companies is bad. I think it is. You can talk about whether the environmental impact of using foundation models is unsustainable and unsupportable. I think, by and large, it is. But that is not to say that statistical inference using convoluted deep neural networks is bad or—and this is where I get into many arguments—that scraping the web to train a convoluted neural network is bad. I think it's fine. Scraping is good, actually.

I think it's very dangerous to say, "The way that we're going to fix the problems we have with AI is to make it illegal to make a record of what's on the Internet." I think that's catastrophic. That's how we never again will know what was on CBS News before it turned into Chud News. Everything Nate Silver ever published on his website was just zeroed out by Disney. You can only see it at the Internet Archive because we scrape. It's just bonkers to say, "It is theft to make transient copies of works, to analyze those transient copies, to publish the results of your analysis."

Those are all socially beneficial activities, and we will all lose if we prohibit them, not least because the firms that creative workers are worried about them, the big media companies, are extremely capable of entering into arrangements with the Big Tech companies to license their corpuses to them in order to try and put us all out of a job. If we get the right to decide who can train an AI with our work, our bosses are just going to modify our contracts to say, "Great, you now must license that right to me. And it's non-negotiable." Failure to learn from that lesson is not tragedy. It is farce.

Rather than ask for a new copyright law, we could make a new labor law, because the only people who've ever beaten AI are the Hollywood screenwriters and actors. And the reason they were able to beat them is because uniquely, among workers in America, they are exempt from the Taft-Hartley Act's prohibition on what's called sectoral bargaining, which is when all the workers in a sector bargain with all the employers in a sector. Now, there are so few workers in America who aren't media workers, who care about copyright, that it rounds to zero, but every single worker in America would benefit from extending sectoral bargaining across the board.

Ars Technica: It could be catastrophic, economically speaking, when the AI bubble finally bursts. But you point out that there might very well be something useful left over when that happens. 

Cory Doctorow: I advise to go long on laser tag arenas because you can definitely turn a data center into one of those. There's not much else you can do with them, unfortunately. A bubble is a way for insiders to pump, and then dump, some mania to the normy investors, to people who've been flushed into the capital markets because they've been denied a defined-benefits pension and who are only really offered market-based pensions. That means you have to be the sucker at the table. You have to put your money into the market if you don't want to die homeless and starving after you retire.

The dot-com bubble was very bad. It separated a lot of pension funds and ordinary investors from their money, but it left behind something very useful. In the early years of the aughts, there was, amid the carnage, quite a liberating vibe where all the stupid money went home and you could get servers for pennies on the dollar. Everybody knew how to code.

A generation of humanities undergraduates were induced to drop out of university and learn Python, Perl, and HTML, and a lot of them were really creative. Your rent dropped by two-thirds in San Francisco. I bought six $1,200 Steelcase Leap chairs, still in the plastic wrap, from a failed dot-com guy on a sidewalk on 19th Street in the Mission District for $25-$50 each and used them as a dining room set for the next 10 years.

So there was a very productive residue that was left behind by the dot-com bubble. It gave rise to a more robust form of the web, Web 2.0, full of things that were more useful, more interesting, more thought-through, more creative, more innovative than the stuff that the bubble threw off in Web 1.0. There are other examples of bubbles that are less likely to throw off that residue. Around that time, we also had Enron. Enron produced nothing, although I do have a pad of Enron stationery that a friend in Austin bought at the bankruptcy auction and sent to me.

So we can distinguish between bubbles with productive residues and unproductive bubbles while still not saying that bubbles are good. Bubbles are bad and destructive. When the cryptocurrency bubble bursts, all that's going to be left are shitty monkey JPEGs and worse Austrian economics. But when AI bursts, you're going to be able to buy GPUs for pennies on the dollar. You're going to have your pick of applied statisticians, many of whom are very creative and have interesting ideas for things you could build with AI but are stuck building the things their bosses want to build. There are going to be these open source models that have barely been touched. Any time someone tries to optimize them, they find so many opportunities to make them run on lower-end and commodity hardware.

DeepSeek was a spin-out of a Chinese hedge fund; the fund gave them $6 million and said, "Go play with these open source models. See what you can squeeze out of them." When they launched, their model was so good running on commodity hardware that the market did a mass sell-off, $600 billion in 24 hours—the largest 24-hour decapitalization of any firm in the history of markets. If you've got cheap hardware and you've got applied statisticians, you've got these open source models and you've got a technology that fundamentally is interesting and has done useful things and will do useful things in the future—that's a better setup than one in which we're all running around arguing about whether the word-guessing program is going to wake up, become God, and turn us into paperclips.

Ars Technica:  You also push back a little on the "AI is coming for your job" messaging.

Cory Doctorow: I think we have to distinguish between the AI doing your job and the AI being incapable of doing your job, but your boss is such a sucker that he fires you and replaces you with the AI anyway. There's infinite evidence for the second one. I think that there's very little evidence for the first one, at least so far. A lot of the stories we've heard, when you interrogate them, just turn out to be nonsense. There's a chapter in the book about how many of the demos for AI have just turned out to be people in India pretending to be robots.

The most egregious example was when Amazon announced that cashiers were now out of a job because now you could just walk into [an Amazon Go store], grab stuff off the shelf, and walk out again, and the AI knows what you took. There wasn't an AI. It was three people in India watching each customer through a network of cameras in the ceiling trying to guess what you put in your bag.

I think there's lots of things that skilled workers will ask AI to do that will help them do their jobs. There's lots of things that skilled workers will ask AI to do that they'll be wrong about and that won't help them do their jobs. And there's probably space at the margin to replace humans with AI, at least in some cases. But the idea that we're at a "jobspocalypse" is such a self-serving narrative. If you're trying to convince people that the way you're going to turn $1.4 trillion in CapEx into more than $1.4 trillion in revenue is by convincing bosses to fire workers and replace them with chatbots, you have to have a story about how the chatbot can do anyone's job.

Here's a wager. If you ever have the opportunity to interview Dario Amodei or Sam Altman, I want you to ask them this. Someday, you will retire. Right now, I want you to make a binding decision. Will the thing that wipes your ass and takes care of you when you are too old and frail to take care of yourself be a person or an AI? We're just going to use whatever it is that's around at that time, and you get to choose. I think we should ask anyone who says they know how to fix things, would they themselves go to an old folk's home run according to the principles they're establishing?

Ars Technica: We are now starting to see news stories about how companies that invested in AI are suddenly getting hefty bills. 

Cory Doctorow: They're getting the bill because the AI companies are trying to get out before they're stuck holding the bag. They want to do IPOs, and to do IPOs, they need to clean up their balance sheet. So they're like, "I bet these [companies] are pretty price-insensitive. Let's just jack it. Let's go from a 90 percent subsidy to a 40 percent subsidy and more than double everyone's prices. They'll hang in there." And then you get the CTO of Uber saying, "I'm not sure why we put AI in the business to begin with, and I really don't know why we'd use it if it was $20,000 a seat. So I don't know that we are going to use AI anymore."

This is quite a backfire. It actually shows you how insulated these people are from any sense of how their products are received, from what people think of them, from the actual fundamentals of real businesses that have to bring in more money than they spend. It's weird. I'm hardly a captain of industry or one of the great champions of markets, but I do understand that, by and large, firms should bring in more money than they spend if they are to be an attractive investment prospect.

Ars Technica: We hear plenty about the negative aspects of AI. What do you like about it?

Cory Doctorow: I have a couple of local models on my computer, which is just a framework laptop running Ubuntu. It doesn't even have a GPU. I use Whisper to transcribe audio. I will sometimes want to cite something I've heard in a podcast and not remember where I heard it. One time, I just threw the last 30 hours of audio I'd listened to at Whisper, and it shot out verbatim logs that were good enough that when I searched the full text, I could find it. And it gave me time codes so I could check the transcript. That's amazing.

The idea that I might someday have a computer full of audio and video files with full text indexing is great. I could even imagine conversational interfaces to that: "Where's the photo of my daughter at her birthday party where she's dressed like a pirate?"

AI doesn't have to be 100 percent accurate for that to be useful. It doesn't have to be free from false positives. It can just be OK. That stuff's running on your own computer. It's not burning down a rainforest. It's not consuming the last three drops of potable water left in Nevada. There is a certain kind of person who is performatively horrified by AI: "But, but, but that's energy you wouldn't have used." I'm like, "You have never said that about someone who turns cell shading on while playing an MMORPG." Everything you do with your computer burns electricity.

I've been using another chatbot where I paste my daily blog post in and say, "Find my typos." It finds a lot of the errors that are normally not caught by a regular spell checker: doubled-up words, punctuation marks, or words that are actual words but are misspellings for other words. When you dial up the sensitivity to the point where it actually catches all of those, it also gets a lot of false positives. That's fine for 1,500 to 3,000 words. I never feed it a book. On a 100,000-word manuscript, it's going to give me thousands of false positives, and it just won't be useful. I treat this like a plugin to my word processor. It's fine. Sometimes it's good, and sometimes it's not.

I have a friend, Patrick Ball, who is the best programmer I know, and he founded and works at an NGO called the Human Rights Data Analysis Group (HRDAG). They're one of the most important NGOs that no one's ever heard of. What they do is very rigorous statistical extrapolations of information that's not in the record, about wars, civil wars, coups, oppressive states, and they're used for truth and reconciliation, human rights tribunals, war crimes trials. They've worked on every high-profile war crimes trial in the century.

Patrick is using a bunch of Copilots to write software to do a lot of special-purpose stuff. For example, they work with Innocence Project of New Orleans, which has exonerated a bunch of [wrongly convicted] people. They can go through all the arrest reports from the New Orleans PD and find the ones that have linguistic correlates that match successful exonerations. Then they give those to the lawyers who would otherwise just be starting alphabetically or chronologically, sorting to the top the ones that are most like the ones that led to a successful exoneration.

It's not like they're asking the chatbot to write a brief for them, but this is a hugely important function, and it is getting innocent people out of prison. You don't want innocent people in prison. That should be the least controversial thing in the world. That's just good, and the proof is in the pudding.

Read full article

Comments

In an extremely odd case, a single 79-year-old patient was granted early access to Eli Lilly's powerful, still-experimental obesity drug retatrutide through the Food and Drug Administration's "compassionate use" program—raising immediate questions if that sole patient is President Donald Trump, according to a report by Stat News.

Lilly's retatrutide is a highly anticipated next-generation obesity drug that targets GIP and glucagon hormones in addition to GLP-1. It is currently in late-stage trials to treat obesity, diabetes, sleep apnea, and other conditions. Data from a Phase 3 trial that Lilly released in May indicates that patients with obesity (but without diabetes) who took the drug for 80 weeks lost 28 percent of their weight, an amount comparable to bariatric surgery.

Millions of Americans with obesity are eager to get the drug, with options being limited so far to enrolling in a clinical trial or trying to obtain it by dodgy methods.

But according to a barebones public notice and Stat's sources, a single person has been granted early access through the expanded access, aka "compassionate use" pathway, which is typically used to grant access to patients with a "serious or immediately life-threatening disease or condition" and who are not able to enroll in a clinical trial, often because they are too ill.

The access request was first made in April, when the person was 79 years old (Trump turned 80 on June 14). It was made by a senior clinician at the National Institutes of Health named Ranganath Muniyappa, who requested it on behalf of a patient with refractory obesity, obstructive sleep apnea, and pulmonary hypertension, which is high blood pressure in the lungs. Sources told Stat this patient had spent a year on tirzepatide, a drug that targets the GLP-1 and GIP hormones. But the patient had achieved only moderate weight loss on the drug.

The patient was not recommended for bariatric surgery, given their age and other conditions. It was unclear whether the person would have been eligible for a trial. It's also unclear if retatrutide would work in patients who have failed to see success with tirzepatide.

"Something very wrong"

The public notice of the expanded access is suspicious, omitting much of the information that such a notice would normally include, such as the conditions that might qualify a patient for such access.

"Only people in the know would be able to find this [notice], using the drug name," Richard Klein, who helped launch the FDA’s expanded access program in the 1980s, told Stat. "There is something very wrong with the way this is listed because no one would know what it is from the listing, or what it’s for."

Stat asked both the White House and the Department of Health and Human Services if Trump is the patient, and if he has obstructive sleep apnea and pulmonary hypertension, which were not included in a memo of his most recent medical evaluation. White House spokesperson Kush Desai did not answer the question and deferred to the health department. HHS spokesperson Emily Hilliard also did not directly deny that Trump is the patient.

She provided a statement saying:

The FDA supports expanded access programs that can provide patients with serious or life-threatening conditions access to investigational treatments when no comparable or satisfying approved therapies are available. Each request is reviewed on a case-by-case basis based on the clinical circumstances and applicable statutory and regulatory requirements.

Over a dozen experts who spoke to Stat said it was highly unusual for a drug company to grant expanded use of a drug for common conditions to a single patient rather than a cohort of patients with a specified profile.

Lilly spokesperson Misty Fuller did not answer Stat's questions, saying, "We make these decisions following all applicable regulations." The NIH clinician who made the request, Muniyappa, also did not respond to questions.

Read full article

Comments

I recently noticed that Blue Cat’s Plug’n Script makes a couple of appearances in Jim Lill’s latest video exploring microphone preamps and in particular Neve consoles’ preamps. For those who …

Stop thinking of delay as an echo. Start thinking of it as a creative system for building entirely new sounds and getting inspired instead. Most delay plug-ins are built to …

WWDC answered whether your assistant is private. It never answered whether it’s telling the truth — and Apple just gave it hands.

The smartest thing I’ve read about Apple’s WWDC didn’t come from Apple. It came from an analyst named Nate B. Jones, who watched the same keynote everyone else did and noticed that the real story wasn’t whether Siri had finally gotten smart. The real story, he argued, is a land grab over what he calls the trusted action surface — the place where AI actually meets your work, touches your apps, and is handed permission to do something. There are two great bottlenecks in AI, he points out: raw compute, which is Jensen Huang’s kingdom, and the trusted surface where intelligence becomes useful, which is the one Apple just went to war for. Whoever owns that surface owns the meter when intelligence becomes unavoidable. It’s a sharp frame, and he’s right.

He’s right, but he left out the scariest part.

Here is what Apple actually did. It tore Siri down to the studs and rebuilt it on Google’s Gemini — reportedly a custom, 1.2-trillion-parameter model that Apple pays Google something on the order of a billion dollars a year to use. It gave the new assistant eyes: real-time awareness of what’s on your screen. And — this is the part that matters — it gave it hands. The new Siri doesn’t just answer. It manages your browser tabs, rewrites your weak passwords, reaches across your apps, pulls context out of Mail and Messages in the middle of a phone call, and acts inside the software where your life actually happens. Craig Federighi wrapped the whole thing in a promise about privacy and took a quiet shot at the rest of the industry for chasing AI for its own sake while losing sight of the people it’s supposed to serve.

It was a good keynote. And it answered exactly one of the two questions that matter.

Apple answered is it private? — will the assistant know your life without strip-mining it and selling the tailings? That’s a real question, and Apple has a real, earned answer. The other question, the one nobody on that stage went near, is is it true? And the instant you give an assistant hands, that second question stops being academic.

Think about what changes when an assistant goes from talking to doing. When a chatbot makes something up, it costs you a minute and a raised eyebrow. When an agent makes something up and then acts on it, it sends the email to the wrong person, moves money to the wrong account, deletes the file it meant to keep, books the flight for the wrong Thursday. A hallucination in a chat window is an annoyance. A hallucination with hands is an incident. Agency multiplies the cost of being wrong by exactly the thing that makes agency valuable.

And what is driving those hands? A large language model — a very capable one, but the same kind of machine that, like every model of its line, fabricates with total confidence when it doesn’t know. Gemini is excellent and Gemini hallucinates; both are true, the way both are true of all of them. Apple took the most capable probabilistic guess-engine it could license, gave it the keys to your apps and permission to act, and then reassured you about privacy. The locks on the doors are magnificent. Nobody mentioned whether the butler tells the truth.

This is the gap I’ve spent this whole series circling, and I’ll disclose again that I co-founded a company built on closing it, so weigh that however you like. But the principle stands on its own, and it is bigger than any one company: a trusted action surface is only as trustworthy as the facts the agent acts on. You can own the device, the operating system, the permission prompt, the whole beautiful surface — and if the thing deciding what to do is guessing, you have built a faster, smoother way to be confidently wrong about someone’s money. The surface needs a substrate. The hands need a conscience. Trust has two axes — privacy and veracity — and at WWDC Apple shipped one of them and didn’t mention the other was missing.

Which loops back to Jensen, and to the argument I’ve been making from the other direction. Nate’s case is that value migrates off the model and onto the surface, and that this is NVIDIA’s problem. Mine has been that value migrates off the GPU and onto a humbler kind of silicon — the CPU —, because most of what we ask AI to do is look something up, not dream. Two different roads, one destination: the belief that NVIDIA’s position is a law of nature is a story, not a fact. But notice that the surface only wins if people trust it — and the agentic surface raises the trust bar at the very instant it raises the stakes. Owning the meter is worthless if the meter lies.

So watch the surfaces, as Nate says. But watch what they’re built on. Apple just gave a billion people’s computers hands, eyes, and access, and wrapped it in the best privacy story in the industry. It gave Siri hands before it gave Siri honesty. Until the conscience ships, the hands are the part I would keep an eye on.

Robert X. Cringely is a co-founder of 2Brains, Inc., in Charlottesville, Virginia. He has written this column since 1987.

The post Apple Gave Siri Hands first appeared on I, Cringely.






Digital Branding
Web Design Marketing

Why the machines hallucinate, why they have no worldview, and why truth has to come from somewhere else.

I’m going to say something that sounds like an insult and is meant as a description: large language models (all of them) hav never known a true thing. Not once. It doesn’t know things at all. It is extraordinarily good at sounding like it does, which is a different skill, and most of our present confusion comes from mistaking the second for the first.

Here is what a language model actually does. It has read an enormous amount of text, and from that text it has learned, with real brilliance, what tends to come next. Give it some words and it predicts the words likely to follow. That is the whole trick. It is a magnificent trick — it gives us machines that write fluent prose in any voice on any subject — but look at what it optimizes for. It optimizes for plausible. It was never, at any point, optimizing for true. Truth was not in the objective. Plausibility was. And plausibility and truth often travel together, which is precisely why we confuse them — but they are not the same thing, and the gap between them is the whole story.

This is why these systems “hallucinate,” a word I dislike because it implies a malfunction. There is no malfunction. A model that invents a court case that never happened — complete with a docket number, plausible parties, and a tidy holding — is not broken. It is doing exactly what it was built to do: produce the most plausible continuation. A fake citation is plausible. It looks like the thousands of real ones the model has read. The machine has no way to prefer the real one, because it has no idea that “real” is a category. It isn’t lying, either. Lying requires knowing the truth and choosing against it, and the machine has never once been in a position to know.

Now the deeper point, the one that took me a long time to learn to say cleanly. Truth is not a property of language. You cannot find it inside a sentence by examining the sentence harder. Truth is a property of the relationship between a sentence and the world — between the words “it is raining” and the actual sky. A statement is true when it corresponds to how things are. And the model has only ever seen the words. It has read every description of rain ever written and stood out in none of it. It holds the map — all of the maps, every map anyone has ever drawn — and it has never once been to the territory. That is why it can be eloquent and wrong in the same breath and feel no friction between the two. The friction lives in a place the model has never visited.

There’s a corollary that unsettles people, and it shouldn’t. A machine like this has no worldview. None. It will argue any side of anything with equal grace, defend a position and then dismantle it in the next window, because it isn’t holding a position — it’s rendering one. It is a mirror with a vocabulary. We keep waiting for it to reveal what it really believes, and it doesn’t believe anything, and that is not a flaw to be trained out of it. It is the honest fact of the thing. The language is separate from any view of the world. That was the original insight some of us started from years ago, before any of the building began: language is machinery, and machinery has no creed.

It is a mirror with vocabulary

The trouble is that we keep dressing the machinery in the costume of a knower. We put it behind a chat window that answers in the first person, warm and certain, and every instinct we have says this thing believes what it is telling me. It does not. It cannot. And the distance between how it sounds and what it is happens to be the most dangerous real estate in the whole technology, because that is exactly where a fluent falsehood gets received as a considered judgment — in a clinic, in a courtroom, in a loan decision, in a room where someone is deciding whether to act.

So what do you do with a machine that can say anything and stand behind nothing? You stop asking it to be the thing it cannot be. If truth lives in the relationship between a claim and the world, then truth has to come from the world — from some grounded, checkable account that sits outside the language model and stays outside it. You don’t teach the renderer to be honest. You keep the saying and the knowing in separate rooms, and you let the language render only what the knowing will vouch for. Language on one side, a verifiable account of the world on the other, and a wall between them you can actually inspect.

That sounds tidy until you try to build it, and then you hit the part nobody puts on a slide. Before you can check a claim against the world, you have to know what the claim is — and pulling discrete, checkable claims out of fluent prose is genuinely hard. The machine doesn’t speak in clean facts. It speaks in paragraphs, where an assertion hides inside a subordinate clause, where a hedge can pass for a claim and a claim can pass for a hedge, and where — my favorite trap — every individual sentence is true and the paragraph they assemble into is a lie. The honest sentence, marshaled into a dishonest whole. Working out what is actually being asserted, before you have checked whether any of it is so, turns out to be most of the labor. It is unglamorous, and it is the ballgame.

I don’t think the future of this technology is a more fluent machine. We already have fluency. Fluent is solved. The future is a more honest architecture — one that knows the difference between what it can say and what it can stand behind, and that keeps the truth somewhere you can point to and check. A machine with no worldview is not the problem. Pretending it has one is. The repair was never going to be giving the machine a conscience. It is to stop asking the part that talks to also be the part that knows.

Full disclosure: I’m a co-founder of 2Brains, a company built on exactly this conviction, so I am not a neutral party here, which we have solved and have patent pending. But the conviction came first. The company exists because of it, not the other way around.

 

The post GenAI is Fluent in Everything, but Faithful in Nothing first appeared on I, Cringely.






Digital Branding
Web Design Marketing